Coordinate-based implicit neural networks, or neural fields, have emerged as useful representations of shape and appearance in 3D computer vision. Despite advances however, it remains challenging to build neural fields for categories of objects without datasets like ShapeNet that provide canonicalized object instances that are consistently aligned for their 3D position and orientation (pose). We present Canonical Field Network (CaFi-Net), a self-supervised method to canonicalize the 3D pose of instances from an object category represented as neural fields, specifically neural radiance fields (NeRFs). CaFi-Net directly learns from continuous and noisy radiance fields using a Siamese network architecture that is designed to extract equivariant field features for category-level canonicalization. During inference, our method takes pre-trained neural radiance fields of novel object instances at arbitrary 3D pose, and estimates a canonical field with consistent 3D pose across the entire category. Extensive experiments on a new dataset of 1300 NeRF models across 13 object categories show that our method matches or exceeds the performance of 3D point cloud-based methods.

Language is one of the primary means by which we describe the 3D world around us. While rapid progress has been made in text-to-2D-image synthesis, similar progress in text-to-3D-shape synthesis has been hindered by the lack of paired (text, shape) data. Moreover, extant methods for text-to-shape generation have limited shape diversity and fidelity. We introduce TextCraft, a method to address these limitations by producing high-fidelity and diverse 3D shapes without the need for (text, shape) pairs for training. TextCraft achieves this by using CLIP and using a multi-resolution approach by first generating in a low-dimensional latent space and then upscaling to a higher resolution, improving the fidelity of the generated shape. To improve shape diversity, we use a discrete latent space which is modelled using a bidirectional transformer conditioned on the interchangeable image-text embedding space induced by CLIP. Moreover, we present a novel variant of classifier-free guidance, which further improves the accuracy-diversity trade-off. Finally, we perform extensive experiments that demonstrate that TextCraft outperforms state-of-the-art baselines.

我们提出了ShapeCrafter，这是一个用于递归文本条件3D形状生成的神经网络。生成文本条件的3D形状的现有方法会消耗整个文本提示，以在一个步骤中生成3D形状。但是，人类倾向于递归描述形状，我们可能以初始描述开始，并根据中间结果逐步添加细节。为了捕获此递归过程，我们引入了一种生成以初始短语为条件的3D形状分布的方法，该方法随着添加更多短语而逐渐发展。由于现有的数据集不足以训练这种方法，因此我们提出了Text2Shape ++，这是一个支持递归形状生成的369K形状文本对的大数据集。为了捕获通常用于完善形状描述的本地细节，我们建立在矢量定量的深层隐式函数的基础上，从而产生高质量形状的分布。结果表明，我们的方法可以生成与文本描述一致的形状，并且随着添加更多短语，形状逐渐发展。我们的方法支持形状编辑，外推，并可以在人机合作中为创意设计提供新的应用程序。

制造物体的3D模型对于填充虚拟世界和视觉和机器人技术的合成数据很重要。为了最有用，应该阐明此类对象：它们的部分应在与之互动时移动。尽管存在铰接式对象数据集，但创建它们是劳动密集型的。基于学习的零件动作预测可以有所帮助，但是所有现有方法都需要带注释的培训数据。在本文中，我们提出了一种无监督的方法，用于发现部分分段的3D形状集合中的铰接运动。我们的方法基于我们称之为闭合的概念：对象的部分的任何有效表达都应将对象保留在同一语义类别中（例如，椅子保持椅子）。我们使用一种算法来实现此概念，该算法优化了形状的零件运动参数，从而可以转换为集合中的其他形状。我们通过使用Partnet-Mobility数据集重新发现零件动作来评估我们的方法。对于几乎所有形状类别，我们方法的预测运动参数在地面真实注释方面的错误较低，表现优于两种监督运动预测方法。

在视觉计算中，3D几何形状以许多不同的形式表示，包括网格，点云，体素电网，水平集和深度图像。每个表示都适用于不同的任务，从而使一个表示形式转换为另一个表示（前向地图）是一个重要且常见的问题。我们提出了全向距离字段（ODF），这是一种新的3D形状表示形式，该表示通过将深度从任何观看方向从任何3D位置存储到对象的表面来编码几何形状。由于射线是ODF的基本单元，因此可以轻松地从通用的3D表示和点云等常见的3D表示。与限制代表封闭表面的水平集方法不同，ODF是未签名的，因此可以对开放表面进行建模（例如服装）。我们证明，尽管在遮挡边界处存在固有的不连续性，但可以通过神经网络（Neururodf）有效地学习ODF。我们还引入了有效的前向映射算法，以转换odf to＆从常见的3D表示。具体而言，我们引入了一种有效的跳跃立方体算法，用于从ODF生成网格。实验表明，神经模型可以通过过度拟合单个对象学会学会捕获高质量的形状，并学会概括对共同的形状类别。

建筑环境中许多物体的形状由他们与人体的关系决定：一个人将如何与这个对象进行互动？ 3D形状的现有数据驱动的生成模型产生合理的物体，但不会理由对人体的那些物体的关系。在本文中，我们学习了3D形状的身体感知生成模型。具体而言，我们培养椅子的生成型号，一种无处不在的形状类别，可以在给定的身体形状或坐姿姿势调节。身体形状调节的型号生产椅子，为具有给定体形的人舒适;姿势调节模型生产适应坐姿的椅子。要训​​练这些模型，我们定义了“坐姿匹配”度量标准和小说“坐姿舒适”度量。计算这些指标需要昂贵的优化将身体置于椅子上，这太慢被用作用于训练生成模型的损耗功能。因此，我们训练神经网络以有效地近似这些度量。我们使用我们的方法培训三个身体感知生成形状模型：基于结构的零件的发电机，点云发生器和隐式表面发生器。在所有情况下，我们的方法都生产适应其输出椅形状以输入人体规格的型号。

机器学习的最近进步已经创造了利用一类基于坐标的神经网络来解决视觉计算问题的兴趣，该基于坐标的神经网络在空间和时间跨空间和时间的场景或对象的物理属性。我们称之为神经领域的这些方法已经看到在3D形状和图像的合成中成功应用，人体的动画，3D重建和姿势估计。然而，由于在短时间内的快速进展，许多论文存在，但尚未出现全面的审查和制定问题。在本报告中，我们通过提供上下文，数学接地和对神经领域的文学进行广泛综述来解决这一限制。本报告涉及两种维度的研究。在第一部分中，我们通过识别神经字段方法的公共组件，包括不同的表示，架构，前向映射和泛化方法来专注于神经字段的技术。在第二部分中，我们专注于神经领域的应用在视觉计算中的不同问题，超越（例如，机器人，音频）。我们的评论显示了历史上和当前化身的视觉计算中已覆盖的主题的广度，展示了神经字段方法所带来的提高的质量，灵活性和能力。最后，我们展示了一个伴随着贡献本综述的生活版本，可以由社区不断更新。

The goal of this paper is to estimate the 6D pose and dimensions of unseen object instances in an RGB-D image. Contrary to "instance-level" 6D pose estimation tasks, our problem assumes that no exact object CAD models are available during either training or testing time. To handle different and unseen object instances in a given category, we introduce Normalized Object Coordinate Space (NOCS)-a shared canonical representation for all possible object instances within a category. Our region-based neural network is then trained to directly infer the correspondence from observed pixels to this shared object representation (NOCS) along with other object information such as class label and instance mask. These predictions can be combined with the depth map to jointly estimate the metric 6D pose and dimensions of multiple objects in a cluttered scene. To train our network, we present a new contextaware technique to generate large amounts of fully annotated mixed reality data. To further improve our model and evaluate its performance on real data, we also provide a fully annotated real-world dataset with large environment and instance variation. Extensive experiments demonstrate that the proposed method is able to robustly estimate the pose and size of unseen object instances in real environments while also achieving state-of-the-art performance on standard 6D pose estimation benchmarks.

We propose a layered hierarchical architecture called UCLA (Universal Causality Layered Architecture), which combines multiple levels of categorical abstraction for causal inference. At the top-most level, causal interventions are modeled combinatorially using a simplicial category of ordinal numbers. At the second layer, causal models are defined by a graph-type category. The non-random ``surgical" operations on causal structures, such as edge deletion, are captured using degeneracy and face operators from the simplicial layer above. The third categorical abstraction layer corresponds to the data layer in causal inference. The fourth homotopy layer comprises of additional structure imposed on the instance layer above, such as a topological space, which enables evaluating causal models on datasets. Functors map between every pair of layers in UCLA. Each functor between layers is characterized by a universal arrow, which defines an isomorphism between every pair of categorical layers. These universal arrows define universal elements and representations through the Yoneda Lemma, and in turn lead to a new category of elements based on a construction introduced by Grothendieck. Causal inference between each pair of layers is defined as a lifting problem, a commutative diagram whose objects are categories, and whose morphisms are functors that are characterized as different types of fibrations. We illustrate the UCLA architecture using a range of examples, including integer-valued multisets that represent a non-graphical framework for conditional independence, and causal models based on graphs and string diagrams using symmetric monoidal categories. We define causal effect in terms of the homotopy colimit of the nerve of the category of elements.

Self-supervised pre-trained transformers have improved the state of the art on a variety of speech tasks. Due to the quadratic time and space complexity of self-attention, they usually operate at the level of relatively short (e.g., utterance) segments. In this paper, we study the use of context, i.e., surrounding segments, during fine-tuning and propose a new approach called context-aware fine-tuning. We attach a context module on top of the last layer of a pre-trained model to encode the whole segment into a context embedding vector which is then used as an additional feature for the final prediction. During the fine-tuning stage, we introduce an auxiliary loss that encourages this context embedding vector to be similar to context vectors of surrounding segments. This allows the model to make predictions without access to these surrounding segments at inference time and requires only a tiny overhead compared to standard fine-tuned models. We evaluate the proposed approach using the SLUE and Librilight benchmarks for several downstream tasks: Automatic speech recognition (ASR), named entity recognition (NER), and sentiment analysis (SA). The results show that context-aware fine-tuning not only outperforms a standard fine-tuning baseline but also rivals a strong context injection baseline that uses neighboring speech segments during inference.